Image processing apparatus for endoscope, image processing method for endoscope, and recording medium

ABSTRACT

An image processing apparatus for endoscope includes a processor. The processor sequentially receives a plurality of observation images of an endoscope, detects one or more lesioned parts from each of the plurality of observation images, analyzes visibility of at least one of a distance from the endoscope, an occupied area, a shape, a size, a position in the observation image, a color, luminance, or an organ part relating to the detected lesioned part, sets a display extension time of a detection result of the lesioned part according to the visibility, and outputs the observation image to which the detection result of the lesioned part is added.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of PCT/JP2018/025791filed on Jul. 6, 2018, the entire contents of which are incorporatedherein by this reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image processing apparatus forendoscope, an image processing method for endoscope, and a recordingmedium.

2. Description of the Related Art

Endoscopes have been widely used in the medical field and the industrialfield. For example, in the medical field, a surgeon can find andidentify a lesioned part by viewing an endoscope image of an inside of asubject displayed on a display apparatus and perform treatment on thelesioned part using a treatment instrument.

An image processing apparatus that adds highlighted display by a markersuch as a frame to a lesioned part detected from an endoscopic image anddisplays the lesioned part in order to prevent a surgeon fromoverlooking the lesioned part when viewing an endoscopic image has beengenerally well known.

Incidentally, in an endoscopic observation, since relative positions ofan object in a body cavity, an image of which is picked up by anendoscope, and an insertion section of the endoscope inserted into thebody cavity can always change, it is difficult to correctly detect, inall frames, a lesioned part once detected. Therefore, a method ofperforming, for a frame in which the lesioned part is not detected,extended display of a marker or the like using information concerning anearest frame in which the lesioned part is detected is conceivable.

For example, it is possible to realize the extended display of themarker or the like by applying, to an image processing apparatus forendoscopic image, a method of performing, for a general image, extendeddisplay of display information, proposed in Japanese Patent ApplicationLaid-Open Publication No. 2009-105705.

SUMMARY OF THE INVENTION

An image processing apparatus for endoscope according to an aspect ofthe present invention includes a processor. The processor sequentiallyreceives a plurality of observation images obtained by picking up animage of an object with an endoscope; detects one or more lesionedparts, which each is an observation target of the endoscope, from eachof the plurality of observation images; analyzes visibility of at leastone of a distance from the endoscope, an occupied area, a shape, a size,a position in each of the observation images, a color, luminance, or anorgan part relating to each of the detected lesioned parts; sets adisplay extension time of a detection result of each of the lesionedparts according to the analyzed visibility; and outputs each of theobservation images to which the detection result of the lesioned partsis added.

An image processing method for endoscope according to an aspect of thepresent invention includes: sequentially receiving a plurality ofobservation images obtained by picking up an image of an object with anendoscope; detecting one or more lesioned parts, which each is anobservation target of the endoscope, from each of the plurality ofobservation images; analyzing visibility of at least one of a distancefrom the endoscope, an occupied area, a shape, a size, a position ineach of the observation images, a color, luminance, or an organ partrelating to each of the detected lesioned parts; setting a displayextension time of a detection result of each of the lesioned partsaccording to the analyzed visibility; and outputting each of theobservation images to which the detection result of the lesioned partsis added.

A recording medium according to an aspect of the present invention is acomputer-readable non-transitory recording medium that stores a computerprogram, the computer program causing the computer to: sequentiallyacquire a plurality of observation images obtained by picking up animage of an object with an endoscope; detect one or more lesioned parts,which each is an observation target of the endoscope, from each of theplurality of observation images; analyze visibility of at least one of adistance from the endoscope, an occupied area, a shape, a size, aposition in each of the observation images, a color, luminance, or anorgan part relating to each of the detected lesioned parts; set adisplay extension time of a detection result of each of the lesionedparts according to the analyzed visibility; and output each of theobservation images to which the detection result of the lesioned partsis added.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing a configuration of a main part of anendoscope system including an image processing apparatus according to afirst embodiment;

FIG. 2 is a block diagram for explaining an example of a configurationrelating to image processing of the image processing apparatus accordingto the first embodiment;

FIG. 3 is a flowchart showing an example of processing performed in theimage processing apparatus according to the first embodiment;

FIG. 4 is a time chart for explaining an example of processing performedin the image processing apparatus according to the first embodiment;

FIG. 5 is a time chart for explaining another example of the processingperformed in the image processing apparatus according to the firstembodiment;

FIG. 6 is a diagram showing an example of an image for display displayedon a display apparatus through processing of the image processingapparatus according to the first embodiment;

FIG. 7 is a diagram showing another example of the image for displaydisplayed on the display apparatus through the processing of the imageprocessing apparatus according to the first embodiment;

FIG. 8 is a diagram showing still another example of the image fordisplay displayed on the display apparatus through the processing of theimage processing apparatus according to the first embodiment;

FIG. 9 is a block diagram for explaining an example of a configurationof a lesion-state analyzing section according to a second embodiment;

FIG. 10 is a flowchart showing an example of a flow of lesion analysisprocessing according to the second embodiment;

FIG. 11 is a flowchart showing an example of a flow of lesion analysisprocessing according to a modification of the second embodiment;

FIG. 12 is a flowchart showing an example of a flow of lesion analysisprocessing according to a third embodiment; and

FIG. 13 is a flowchart showing an example of a flow of lesion analysisprocessing according to a modification of the third embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention are explained below with referenceto the drawings.

First Embodiment

FIG. 1 is a diagram showing a configuration of a main part of anendoscope system including an image processing apparatus according to afirst embodiment. An endoscope system 1 includes, as shown in FIG. 1 , alight-source driving apparatus 11, an endoscope 21, a video processor31, an image processing apparatus for endoscope (hereinafter referred toas image processing apparatus) 32, and a display apparatus 41.

The light-source driving apparatus 11 includes, for example, a drivecircuit. The light-source driving apparatus 11 is connected to theendoscope 21 and the video processor 31. The light-source drivingapparatus 11 is configured to generate, based on a light source controlsignal from the video processor 31, a light source driving signal fordriving a light source section 23 of the endoscope 21 and output thegenerated light source driving signal to the endoscope 21.

The endoscope 21 is connected to the light-source driving apparatus 11and the video processor 31. The endoscope 21 includes anelongated-shaped insertion section 22 insertable into a body cavity of asubject. The light source section 23 and an image pickup section 24 areprovided at a distal end portion of the insertion section 22.

The light source section 23 includes a light emitting element such as awhite LED. The light source section 23 is configured to generateillumination light by emitting light according to the light sourcedriving signal outputted from the light-source driving apparatus 11 andemit the generated illumination light to an object such as a biologicaltissue.

The image pickup section 24 includes an image sensor such as a color CCDor a color CMOS. The image pickup section 24 is configured to performoperation corresponding to an image pickup control signal outputted fromthe video processor 31. The image pickup section 24 is configured toreceive reflected light from the object illuminated by the illuminationlight from the light source section 23, pick up an image of the receivedreflected light to generate an image pickup signal, and output thegenerated image pickup signal to the video processor 31.

The video processor 31 is connected to the light-source drivingapparatus 11 and the endoscope 21. The video processor 31 is configuredto generate a light source control signal for controlling a lightemission state of the light source section 23 and output the lightsource control signal to the light-source driving apparatus 11. Thevideo processor 31 is configured to generate an image pickup controlsignal for controlling an image pickup operation of the image pickupsection 24 and output the image pickup control signal. The videoprocessor 31 is configured to generate an observation image G1 of anobject by applying predetermined processing to an image pickup signaloutputted from the endoscope 21 and sequentially output the generatedobservation image G1 to the image processing apparatus 32 frame byframe.

The image processing apparatus 32 includes an electronic circuit such asan image processing circuit. The image processing apparatus 32 isconfigured to perform operation for generating an image for displaybased on the observation image G1 outputted from the video processor 31and causing the display apparatus 41 to display the generated image fordisplay. The image processing apparatus 32 includes, as shown in FIG. 2, an image input section 33, a lesion detecting section 34, and adisplay control section 36. FIG. 2 is a block diagram for explaining anexample of a configuration relating to image processing of the imageprocessing apparatus according to the first embodiment.

Note that the image processing apparatus 32 is not limited to beconfigured by the processor including the electronic circuit such as theimage processing circuit. For example, the image processing apparatus 32may be configured to serve functions of the respective sections in theimage processing apparatus 32 by causing a processor including a CPU toexecute software. The image processing apparatus 32 may be configured bya processor including an integrated circuit such as an FPGA (fieldprogrammable gate array) including circuit sections corresponding to therespective sections in the image processing apparatus 32. A computerprogram to be executed by a computer (software to be executed by aprocessor) is recorded in a computer-readable non-transitory recordingmedium.

The image input section 33 outputs the observation image G1 receivedfrom the video processor 31 to the lesion detecting section 34.

The lesion detecting section 34 is configured to detect a lesion regionLn (n=1, 2, . . . ) included in the observation image G1 sequentiallyoutputted from the image input section 33. The lesion detecting section34 detects the lesion region Ln from the observation image G1 byperforming processing for applying, to the observation image G1, animage identifier that acquires, in advance, a function capable ofidentifying a polyp image with a learning method such as deep learning.Note that the detection of the lesion region Ln is not limited to thelearning method described above and other methods may be used. Forexample, polyp candidate detection processing disclosed in JapanesePatent Application Laid-Open Publication No. 2007-244518 may be used.

The display control section 36 functioning as a display-control outputsection is connected to the display apparatus 41. The display controlsection 36 includes a highlighting processing section 36 a, alesion-state analyzing section 36 b, a display-extension-time settingsection 36 c, and a recording section 36 d. The display control section36 is configured to perform processing for generating an image fordisplay using the observation image G1 sequentially outputted from thevideo processor 31 and perform processing for causing a display screen41A of the display apparatus 41 to display the generated image fordisplay.

In order to highlight a position of the lesion region Ln detected by thelesion detecting section 34, the highlighting processing section 36 aperforms highlighting processing for generating a marker image G2surrounding the lesion region Ln and adding the marker image G2 to theobservation image G1. The highlighting processing is started from apoint in time when the lesion region Ln is detected. The highlightingprocessing ends when a display extension time set by thedisplay-extension-time setting section 36 c elapses after the detectionof the lesion region Ln is interrupted. Note that the interruptionincludes a case in which the lesion region Ln moves to an outside of ascreen after the lesion region Ln is detected and a case in which thelesion detecting section 34 fails in the detection of the lesion regionLn.

Note that the marker image G2 added by the highlighting processing ofthe highlighting processing section 36 a may have any form as long asthe marker image G2 is capable of presenting the position of the lesionregion Ln as visual information. The marker image G2 may be any imagehaving, for example, a square shape, a triangular shape, a circularshape, and a star shape. The marker image G2 may be an image notsurrounding the lesion region Ln if the image can indicate the positionof the lesion region Ln. For example, the position of the lesion regionLn may be indicated by differentiating brightness and a color tone ofthe lesion region Ln from brightness and a color tone of a peripheralregion. Further, presence of a lesion region may be indicated bygenerating a message indicating the lesion region as support informationand displaying the message in a form of a popup message or the like in avicinity of the lesion region, a periphery of an observation image, orthe like or generating and displaying a flag.

The lesion-state analyzing section 36 b analyzes a state of the lesionregion Ln detected by the lesion detecting section 34. A result of theanalysis is outputted to the display-extension-time setting section 36c.

The display-extension-time setting section 36 c sets, based on theanalysis result of the state of the lesion region Ln, a displayextension time for the marker image G2 generated by the highlightingprocessing section 36 a. Note that the display extension time set by thedisplay-extension-time setting section 36 c means a time period in whichthe marker image G2 additionally displayed in the observation image G1is displayed in the image for display even after the detection of thelesion region Ln is interrupted. Two or more kinds of display extensiontimes are set in advance in the display-extension-time setting section36 c. An appropriate display extension time is selected according to ananalysis result received from the lesion-state analyzing section 36 b.

The display extension time is specified according to the number offrames. For example, one frame is set in advance as a first displayextension time and ten frames are set in advance as a second displayextension time. When the number of frames per one second is thirty, thefirst display extension time is approximately 0.03 seconds and thesecond display extension time is approximately 0.33 seconds.

In a state in which the lesion region Ln is less easily seen as a resultof analyzing the state of the lesion region Ln or when malignancy ishigh, the second display extension time is selected in order to preventthe surgeon from overlooking a lesioned part. On the other hand, in astate in which the lesion region Ln is easily seen or when malignancy islow, the first display extension time is selected in order to improvevisibility.

The recording section 36 d is configured to sequentially record (in timeseries), as a plurality of recorded images R1, a plurality ofobservation images G1 sequentially outputted from the video processor31. Note that the recording section 36 d may also function as acomputer-readable non-transitory recording medium that stores a computerprogram.

The display apparatus 41 includes a monitor or the like and isconfigured to be able to display an image for display outputted from theimage processing apparatus 32.

Subsequently, action of the present embodiment is explained. FIG. 3 is aflowchart showing an example of processing performed in the imageprocessing apparatus according to the first embodiment. Note that thecomputer program recorded in the recording medium may be a program forperforming processing shown in FIG. 3 (and FIG. 10 to FIG. 13 referredto below).

For example, when power supplies of the light-source driving apparatus11 and the video processor 31 are turned on, the endoscope 21 emitsillumination light to an object, receives reflected light from theobject, picks up an image of the received reflected light to generate animage pickup signal, and outputs the generated image pickup signal tothe video processor 31.

The video processor 31 generates the observation image G1 of the objectby applying predetermined processing to the image pickup signaloutputted from the endoscope 21 and sequentially outputs the generatedobservation image G1 to the image processing apparatus 32 frame byframe. In other words, the image input section 33 acquires an endoscopicimage (the observation image G1), which is an in-vivo intraluminalimage, from the video processor 31 (S1). The image input section 33outputs the acquired image to the lesion detecting section 34.

The lesion detecting section 34 detects the lesion region Ln from theobservation image G1 by performing processing for applying, to theobservation image G1, an image identifier that acquires, in advance, afunction capable of identifying a polyp image with a learning methodsuch as deep learning (S2). A detection result of the lesion region Lnis outputted to the display control section 36.

In the display control section 36, the lesion-state analyzing section 36b analyzes a state of the lesion region Ln. The lesion-state analyzingsection 36 b determines importance (malignancy or the like), a position,and the like of the lesion region Ln with an image analysis andevaluates, for example, possibility of overlooking (S3). Thedisplay-extension-time setting section 36 c receives an analysis resultof the state of the lesion region Ln.

Subsequently, the display-extension-time setting section 36 c sets,based on the analysis result of the state of the lesion region Ln, adisplay extension time for the marker image G2 added to the displayscreen in order to highlight the lesion region Ln (S4). In a state inwhich the lesion region Ln is less easily seen or when malignancy ishigh (S4, YES), the processing proceeds to S6 and thedisplay-extension-time setting section 36 c sets the display extensiontime long. On the other hand, in a state in which the lesion region Lnis easily seen or when malignancy is low (S4, NO), the processingproceeds to S5 and the display-extension-time setting section 36 c setsthe display extension time short.

When the setting of the display extension time ends according to theprocessing of S5 or S6, the processing proceeds to processing of S7. InS7, the highlighting processing section 36 a performs highlightingprocessing for generating the marker image G2 surrounding the lesionregion Ln and adding the marker image G2 to the observation image G1.The highlighting processing is started from a point in time when thelesion region Ln is detected and ends at a point in time when thedisplay extension time elapses after the detection of the lesion regionLn is interrupted because the lesion region Ln moves to an outside of ascreen or the lesion detecting section 34 fails in the detection. Thedisplay control section 36 outputs an image obtained by adding themarker image G2 to the observation image G1 according to necessity tothe display apparatus 41 as an image for display and ends a series ofprocessing.

FIG. 4 and FIG. 5 are time charts for explaining an example ofprocessing performed in the image processing apparatus according to thepresent embodiment. FIG. 4 is a time chart in a state in which thelesion region Ln is less easily seen or in a case in which malignancy ishigh. FIG. 5 is a time chart in a state in which the lesion region Ln iseasily seen or a case in which malignancy is low. FIG. 6 is a diagramshowing an example of an image for display displayed on the displayapparatus through processing of the image processing apparatus accordingto the present embodiment.

At timing (=time point Ta) when detection of a lesion region L1 by thelesion detecting section 34 is started, the display control section 36performs processing for causing the display apparatus 41 to display animage for display in which the observation image G1 obtained by addingthe marker image G2 to the lesion region L1 is arranged in a displayregion D1 of the display screen 41A. With such operation of the displaycontrol section 36, for example, in a period of time Ta to Tb shown inFIG. 4 and FIG. 5 , an image for display shown in FIG. 6 is displayed onthe display screen 41A of the display apparatus 41.

The display control section 36 continues the addition of the markerimage G2 to the observation image G1 in a period from timing (=timepoint Tb) when the detection of the lesion region L1 by the lesiondetecting section 34 is interrupted to timing (=time point Tc or Td)when the display extension time elapses.

In other words, in a time period TL in which the lesion detectingsection 34 continues the detection of the lesion region L1, the imagefor display in the state in which the marker image G2 is added to theobservation image G1 is generated and displayed irrespective of a stateof the lesion region Ln. After the timing when the detection of thelesion region L1 is interrupted, an addition time of the marker image G2changes according to the state of the lesion region Ln.

In other words, in a state in which the lesion region Ln is less easilyseen or when malignancy is high, the image for display in the state inwhich the marker image G2 is added to the observation image G1 isgenerated and displayed on the display screen 41A of the displayapparatus 41 in a display extension time TD1 from the timing when thedetection of the lesion region L1 is interrupted. On the other hand, ina state in which the lesion region Ln is easily seen or when malignancyis low, the image for display in the state in which the marker image G2is added to the observation image G1 is generated and displayed on thedisplay screen 41A of the display apparatus 41 in a display extensiontime TD2 (TD2<TD1) from the timing when the detection of the lesionregion L1 is interrupted.

In this way, according to the embodiment explained above, by adjusting,according to a state of the legion region Ln, a time period (a displayextension time) in which the marker image G2 is continuously displayedafter the detection of the lesion region Ln is interrupted, it ispossible to reduce overlooking of a lesioned part that could occur in anendoscopic observation and improve visibility.

Note that although the appropriate display extension time is selectedout of the two kinds of times TD1 and TD2 set in advance in the aboveexplanation, the appropriate display extension time may be selected outof three or more kinds of display extension times according to the stateof the lesion region Ln.

Although the image for display includes the observation image G1, whichis a movie, in the above explanation, the image for display may beformed by the observation image G1 and the recorded image R1, which is astill image.

FIG. 7 is a diagram showing another example of the image for displaydisplayed on the display apparatus through the processing of the imageprocessing apparatus according to the present embodiment. When thedetection of the lesion region L1 is interrupted, the display controlsection 36 starts, at timing when the detection of a lesion region L1 bya lesion-candidate detecting section 34 b is interrupted, processing forcausing the display apparatus 41 to display an image for display inwhich the observation image G1 is arranged in the display region D1 ofthe display screen 41A and the recorded image R1 recorded by therecording section 36 d during the detection of the lesion region L1 isarranged in a display region D2 of the display screen 41A. At timingwhen the display extension time elapses from the timing when thedetection of the lesion region L1 is interrupted, the display controlsection 36 ends the processing for arranging the recorded image R1 inthe display region D2 of the display screen 41A.

With such operation of the display control section 36, for example,until the display extension time elapses after the timing (=time pointTb) when the detection of the lesion region L1 is interrupted, the imagefor display in which the recorded image R1 equivalent to the observationimage G1 shown in FIG. 6 is arranged in the display region D2 isdisplayed on the display screen 41A of the display apparatus 41. Notethat it is assumed that, for example, the display region D2 is set inadvance as a region having a size smaller than the display region D1 onthe display screen 41A.

In other words, by indicating the position of the lesion region Ln tothe surgeon even after the detection of the lesion region L1 isinterrupted using a sub-screen as explained above, it is possible tofurther reduce overlooking of a lesioned part while preventingdeterioration in visibility for the observation image G1.

As shown in FIG. 8 , in the display region D2 of the display screen 41A,not only the recorded image R1 but also the marker image G2 may be addedto the lesion region L1 of the image and displayed. FIG. 8 is a diagramshowing still another example of the image for display displayed on thedisplay apparatus through the processing of the image processingapparatus according to the present embodiment. By applying thehighlighting processing and adding the marker image G2 to the recordedimage R1 as well and displaying the recorded image R1, the position ofthe lesioned part is more easily identified. It is possible to furtherreduce overlooking.

Second Embodiment

In the first embodiment explained above, the time period (the displayextension time) in which the marker image G2 is continuously displayedafter the detection of the lesion region Ln is interrupted is adjustedaccording to the state of the lesion region Ln. However, in the presentembodiment, visibility of the lesion region Ln is analyzed and a displayextension time is determined based on a result of the analysis.

An image processing apparatus in the present embodiment has the sameconfiguration as the configuration of the image processing apparatus 32in the first embodiment. The same components are denoted by the samereference numerals and signs and explanation of the components isomitted.

FIG. 9 is a block diagram for explaining an example of a configurationof a lesion-state analyzing section according to the second embodiment.Note that FIG. 9 shows not only a configuration according to the presentembodiment explained below but also a configuration according to a thirdembodiment explained after the present embodiment. Operations ofrespective sections are explained in detail in corresponding parts ofthe following explanation.

FIG. 10 is a flowchart showing an example of a flow of lesion analysisprocessing according to the second embodiment. A visibility analyzingsection 361, in particular, a lesion-unit-information analyzing section361A in the lesion-state analyzing section 36 b in FIG. 9 relates toprocessing shown in FIG. 10 .

The lesion-state analyzing section 36 b analyzes the visibility of thelesion region Ln. First, the lesion-state analyzing section 36 b selectsan item for which an analysis of visibility is performed (S11). Examplesof analysis items of visibility include items such as (a) a distancebetween the endoscope 21 and the lesion region Ln, (b) a ratio of anoccupied area of the lesion region Ln in the observation image G1, (c) ashape of the lesion region Ln, (d) a size of the lesion region Ln, (e) aposition of the lesion region Ln in the observation image G1, (f) acolor and luminance of the lesion region Ln, and (g) an organ part wherethe lesion region Ln is located. The lesion-state analyzing section 36 bperforms an analysis about an item selected out of these items. Notethat the analysis of visibility may be performed by selecting only oneitem or may be performed by selecting a plurality of items.

(a) Distance Between Endoscope 21 and Lesion Region Ln

When this item is selected as the analysis item, processing (S12)explained below is performed. A lesion-distance estimating section 361A1shown in FIG. 9 relates to the processing of S12.

The lesion-distance estimating section 361A1 estimates image pickupdistances to respective pixels in an image. Among publicly-known varioustechniques, photographing distance estimation performed assuming that aphotographing target is a uniform diffuser based on an image isexplained.

More specifically, first, as a low absorption wavelength component, alow absorption wavelength (for example, red (R) wavelength) componenthaving a lowest degree of absorption or diffusion in an organism isselected. This is to reduce a pixel value decrease due to a blood vesselor the like reflected on a mucosal surface and obtain pixel valueinformation most correlating to an image pickup distance to the mucosalsurface and is because, in an image formed by three components of red(R), green (G), and blue (B), the red (R) component is a component of awavelength separating from a blood absorption band and of a longwavelength and is less easily affected by the absorption or thediffusion in the organism. The red (R) component is selected.

The lesion-distance estimating section 361A1 estimates an image pickupdistance obtained by assuming a uniform diffuser based on a pixel valueof the low absorption wavelength component. More specifically, the imagepickup distance is calculated by the following Equation (1).

$\begin{matrix}\lbrack {{{Equatio}n}\mspace{14mu} 1} \rbrack & \; \\{r = \sqrt{\frac{I \times K \times {cos\theta}}{L}}} & {(1)\mspace{14mu}{Equation}}\end{matrix}$

where, r indicates the image pickup distance, I indicates radiationintensity of a light source obtained by measurement beforehand, and Kindicates a diffusion reflection coefficient of the mucosal surface,which is an average value, measured beforehand. θ indicates an angleformed by a normal vector of the mucosal surface and a vector from thesurface to a light source and is a value determined by a positionalrelation between a light source in an insertion section distal endportion of the endoscope 21 and the mucosal surface. An average value isset as θ beforehand. L indicates an R component value of a pixel onwhich the mucosal surface of the image pickup distance estimation targetis reflected.

Note that before the image pickup distance estimation, correction ofpixel value unevenness due to an optical system and an illuminationsystem, which could be accuracy deterioration factors of respectivekinds of processing, and exclusion of non-mucous regions such asspecular reflection, residues, and bubbles may be performed.

A method based on an image is explained above. Besides, the image pickupdistance may be calculated based on a range finding sensor or the like.

As explained above, the lesion-distance estimating section 361A1estimates the distance between the endoscope 21 and the lesion region Lnand outputs the distance as the analysis result.

(b) Ratio of Occupied Area of Lesion Region Ln in Observation Image G1.

When this items is selected as the analysis item, processing (S13)explained below is performed. A lesion-occupied-area calculating section361A2 in FIG. 9 relates to the processing of S13.

The lesion-occupied-area calculating section 361A2 calculates a ratio ofan area occupied by the lesion region Ln in the observation image G1.More specifically, the lesion-occupied-area calculating section 361A2detects the lesion region Ln from the observation image G1 by performingprocessing for applying, to the observation image G1, an imageidentifier that acquires, in advance, a function capable of identifyinga polyp image with a learning method such as deep learning. Thelesion-occupied-area calculating section 361A2 calculates the number ofpixels included in the lesion region Ln. Finally, thelesion-occupied-area calculating section 361A2 calculates an occupiedarea ratio by dividing the number of pixels included in the legionregion Ln by a total number of pixels forming the observation image G1.

Note that position information of the lesion region Ln may be acquiredfrom the lesion detecting section 34. The detection of the lesion regionLn is not limited to the learning method described above and othermethods may be used. For example, polyp candidate detection processingdisclosed in Japanese Patent Application Laid-Open Publication No.2007-244518 may be used.

As explained above, the lesion-occupied-area calculating section 361A2calculates the ratio of the occupied area of the lesion region Ln in theobservation image G1 and outputs the ratio as an analysis result.

(c) Shape of Lesion Region Ln

When this item is selected as the analysis item, processing (S14)explained below is performed. A lesion-shape analyzing section 361A3shown in FIG. 9 relates to the processing of S14.

The lesion-shape analyzing section 361A3 performs identificationclassification based on a shape of the lesioned part. More specifically,the lesion-shape analyzing section 361A3 creates a mask image indicatinga lesion region and calculates a shape feature value based on the image.The shape feature value is classified into, using a classifier such asan SVM, one of a plurality of classes generated by machine learning. Asthe shape feature value, a publicly-known parameter such as circularity,moment, or fractal dimension is used.

For example, in the case of a large intestine polyp, there are anelevated type (I type) and a superficial type (II type). As the elevatedtype, there are sessile (Is) without a constriction in a rising part,sub-sessile (Isp) with a constriction in a rising part, and pedunculate(Ip) with a peduncle. In the superficial type, the large intestine polypis classified into an elevated type (IIa), a flat type (IIb), and adepressed type (IIc).

For example, in the case of a stomach polyp, there are a submucosaltumor type (an elevated I type), a sessile type (an elevated II type), asub-sessile type (an elevated III type), and a pedunculate type (anelevated IV type). For example, in the case of a stomach cancer, thestomach cancer is classified into a superficial type (a 0 type), atumorous type (a 1 type), an ulcerative and localized type (a 2 type),an infiltrative ulcerative type (a 3 type), and a diffuse infiltrativetype (a 4 type), and the like.

As explained above, the lesion-shape analyzing section 361A3 identifiesa shape of the lesioned part and outputs the shape as an analysisresult.

(d) Size of Lesion Region Ln

When this item is selected as the analysis item, processing (S15)explained below is performed. A lesion-size estimating section 361A4shown in FIG. 9 relates to the processing of S15.

First, the lesion-size estimating section 361A4 estimates image pickupdistances to respective pixels in an image. The lesion-size estimatingsection 361A4 may perform the estimation of the image pickup distancesusing the method explained above or the like. The lesion-distanceestimating section 361A1 may perform the processing and acquires aresult.

Subsequently, the lesion-size estimating section 361A4 provides athreshold smaller than an image pickup distance of a pixel near a lesionand a threshold larger than the image pickup distance and extracts,through processing by using the thresholds, a region of an image pickupdistance zone where the lesion is present. The lesion-size estimatingsection 361A4 calculates circularity of the region and, when thecircularity is larger than a predetermined value, detects the region asa lumen.

Finally, the lesion-size estimating section 361A4 compares the lumen andthe lesioned part and estimates a size of the lesioned part.

More specifically, the lesion-size estimating section 361A4 estimates anactual size of the lesion by calculating a ratio occupied by length ofthe lesion with respect to a circumferential length of the detectedlumen. Note that it is also possible to set circumferential lengths oflumens in respective organ parts (positions) beforehand based on anatomyand improve accuracy of size estimation. For example, in the case of alarge intestine examination, estimation of a part (position) of alesioned part of a large intestine may be performed based on aninsertion amount of the insertion section to improve the accuracy of thesize estimation of the actual size of the lesion based on the ratiooccupied by the length of the lesion with respect to the circumferentiallength of the lumen set beforehand in the part (position) of theestimated lesioned part.

As explained above, the lesion-size estimating section 361A4 estimatesthe size of the lesioned part in comparison with the circular size ofthe lumen photographed in the endoscopic image and outputs the size ofthe lesioned part as an analysis result.

(e) Position of Lesion Region Ln in Observation Image G1

When this item is selected as the analysis item, processing (S16)explained below is performed. A lesion-position analyzing section 361A5shown in FIG. 9 relates to the processing of S16.

First, the lesion-position analyzing section 361A5 detects the lesionregion Ln from the observation image G1 and acquires positioninformation by performing processing for applying, to the observationimage G1, an image identifier that acquires, in advance, a functioncapable of identifying a polyp image with a learning method such as deeplearning. Note that the detection of the lesion region Ln is not limitedto the learning method explained above and other methods may be used.For example, polyp candidate detection processing disclosed in JapanesePatent Application Laid-Open Publication No. 2007-244518 may be used.The position information of the lesion region Ln may be acquired fromthe lesion detecting section 34 or the lesion-occupied-area calculatingsection 361A2.

Subsequently, the lesion-position analyzing section 361A5 analyzes aposition of the lesion region Ln in the observation image G1. An exampleof a specific method is explained below. First, the observation image G1is equally divided into three in a vertical direction, and equallydivided into three in the horizontal direction, thus divided into nineblocks. For example, when the observation image G1 includes 1920×1080pixels, when the upper left of the image is assumed to be an origin (0,0), the lesion-position analyzing section 361A5 divides the observationimage G1 into a region (1A) of (0, 0) to (640, 360), a region (1B) of(641, 0) to (1280, 360), a region (1C) of (1281, 0) to (1920, 360), aregion (2A) of (0, 361) to (640, 720), a region (2B) of (641, 361) to(1280, 720), a region (2C) of (1281, 361) to (1920, 720), a region (3A)of (0, 721) to (640, 1080), a region (3B) of (641, 721) to (1280, 1080),and a region (3C) of (1281, 721) to (1920, 1080).

The lesion-position analyzing section 361A5 specifies, among the nineblocks 1A to 3C, a block where the lesion region Ln is present andoutputs the block as a position of the lesion region Ln. Note that whenthe legion region Ln is present across a plurality of blocks, a blockwhere an area of presence of the lesion region Ln is the largest is setas the block where the lesion region Ln is present. Note that a methodof specifying the block where the lesion region Ln is present is notlimited to the method explained above. Other methods may be used such asa method of setting, as the block where the lesion region Ln is present,a block where a pixel located in a center of the lesion region Ln ispresent. The number of blocks generated by dividing the observationimage G1 is not limited to nine and may be, for example, 2×2=4 blocks or4×4=16 blocks.

The position of the lesion region Ln may not be the block positionexplained above and may be calculated as a distance from a center pixelposition of the observation image G1.

As explained above, the lesion-position analyzing section 361A5specifies the position of the lesion region Ln in the observation imageG1 and outputs the position as an analysis result.

(f) Color and Luminance of Lesion Region Ln

When this item is selected as the analysis item, processing (S17)explained below is performed. A lesion-color/luminance analyzing section361A6 shown in FIG. 9 relates to the processing of S17.

When the observation image G1 is an image formed by three components ofred (R), green (G), and blue (B), the lesion-color/luminance analyzingsection 361A6 extracts pixel values (R pixel values, G pixel values, andB pixel values) of respective pixels included in the lesion region Ln.The lesion-color/luminance analyzing section 361A6 calculates an averageof each of the R pixel values, the G pixel values, and the B pixelvalues and sets the average as a color pixel value of the lesion regionLn. Note that other statistical values such as a mode may be used forcalculation of a pixel value of the lesion region Ln rather than theaverage.

The lesion-color/luminance analyzing section 361A6 extracts luminancevalues of the respective pixels included in the lesion region Ln,calculates an average value of the luminance values, and sets theaverage value as a luminance value of the lesion region Ln. Note thatother statistical values such as a mode may be used for calculation of aluminance value of the lesion region Ln rather than the average.

As explained above, the lesion-color/luminance analyzing section 361A6calculates the color pixel value and the luminance value of the lesionregion Ln in the observation image G1 and outputs the color pixel valueand the luminance value as an analysis result.

(g) Organ Part where Lesion Region Ln is Located

When this item is selected as the analysis item, processing (S18)explained below is performed. An organ-part analyzing section 361A7shown in FIG. 9 relates to the processing of S18.

The organ-part analyzing section 361A7 performs estimation of anobservation part. For example, when an organ to be observed is a largeintestine, the organ-part analyzing section 361A7 recognizes a rectum, asigmoid colon, a descending colon, a left flexure of colon (a splenicflexure), a transverse colon, a right flexure of colon (a hepaticflexure), an ascending colon, and a caecum. When the organ to beobserved is a stomach, the organ-part analyzing section 361A7 recognizesa cardiac orifice, a stomach fundus, a gastric corpus, a gastric angle,a vestibule, a pyloric region, a pylorus, and a duodenum. In the case ofa small intestine, the organ-part analyzing section 361A7 recognizes ajejunum and an ileum. In the case of an esophagus, the organ-partanalyzing section 361A7 recognizes a cervical esophagus, a thoracicesophagus, and an abdominal esophagus. More specifically, the organ-partanalyzing section 361A7 can perform part (position) estimation using anSVM or the like by collecting image data in which the large intestine,the stomach, the small intestine, and the esophagus are photographed andperforming machine learning using the image data.

As explained above, the organ-part analyzing section 361A7 estimates theobservation part and outputs the observation part as an analysis result.

When the processing of the selected one or more items among theprocessing of S12 to the processing of S18 ends, the lesion-stateanalyzing section 36 b determines visibility based on analysis resultsof these kinds of processing (S19). First, the lesion-state analyzingsection 36 b determines visibility for each of the analysis items.

(a) Distance Between Endoscope 21 and Lesion Region Ln.

When the distance between the endoscope 21 and the lesion region Lnoutputted from the lesion-distance estimating section 361A1 as theanalysis result is larger than a predetermined distance set in advance,the lesion-state analyzing section 36 b determines that the visibilityis low. On the other hand, when the distance between the endoscope 21and the lesion region Ln is equal to or smaller than the predetermineddistance, the lesion-state analyzing section 36 b determines that thevisibility is high.

(b) Ratio of Occupied Area of Lesion Region Ln in Observation Image G1

When the occupancy ratio of the lesion region Ln outputted from thelesion-occupied-area calculating section 361A2 as an analysis result isequal to or lower than a predetermined ratio (for example, 5 percent)set in advance, the lesion-state analyzing section 36 b determines thatthe visibility is low. On the other hand, when the occupancy ratio ofthe lesion region Ln is higher than the predetermined ratio, thelesion-state analyzing section 36 b determines that the visibility ishigh.

(c) Shape of Lesion Region Ln

When the shape of the lesion region Ln outputted from the lesion-shapeanalyzing section 361A3 as the analysis result corresponds to a shapewith high visibility set in advance, the lesion-state analyzing section36 b determines that the visibility is high. For example, shapesdescribed below are set as the shape with high visibility.

When the lesioned part is a large intestine: a superficial flat type(IIb) and a superficial depressed type (IIc).

When the lesioned part is a stomach: a submucosal tumor type (anelevated I type).

When the shape of the lesion region Ln corresponds to a shape with lowvisibility set in advance, the lesion-state analyzing section 36 bdetermines that the visibility is low. For example, shapes describedbelow are set as the shape with low visibility.

When the lesioned part is a large intestine: sessile (Is), sub-sessile(Isp), pedunculate (Ip), and a superficial elevated type (IIa).

When the lesioned part is a stomach: a sessile type (an elevated IItype), a sub-sessile type (an elevated III type), a pedunculate type (anelevated IV type), a tumorous type (a 1 type), an ulcerative andlocalized type (a 2 type), an infiltrative ulcerative type (a 3 type),and a diffuse infiltrative type (a 4 type).

(d) Size of Lesion Region Ln

When the size of the lesion region Ln outputted from the lesion-sizeestimating section 361A4 as the analysis result is equal to or smallerthan a predetermined size (for example, 5 mm) set in advance, thelesion-state analyzing section 36 b determines that the visibility islow. On the other hand, when the size of the lesion region Ln is largerthan the predetermined size, the lesion-state analyzing section 36 bdetermines that the visibility is high.

(e) Position of Lesion Region Ln in Observation Image G1

The lesion-state analyzing section 36 b determines visibility accordingto the position of the lesion region Ln outputted from thelesion-position analyzing section 361A5 as the analysis result. In otherwords, when the position of the lesion region Ln is apart from the imagecenter, the lesion-state analyzing section 36 b determines that thevisibility is low. On the other hand, when the position of the lesionregion Ln is close to the image center, the lesion-state analyzingsection 36 b determines that the visibility is high.

For example, when the block position where the lesion region Ln ispresent is outputted as the analysis result, the lesion-state analyzingsection 36 b sets, as a determination result, visibility registered foreach of blocks in advance. More specifically, it is assumed that, forexample, the observation image G1 is divided into 3×3=9 blocks, fourblocks (blocks 1A, 3A, 1C, and 3C) located at four corners of an imageare registered as having low visibility, and the other five blocks(blocks 2A, 1B, 2B, 3B, and 2C) are registered as having highvisibility. In this case, if the block position where the lesion regionLn is present is the block 1A, 3A, 1C, or 3C, the lesion-state analyzingsection 36 b determines that the visibility is low. If the blockposition where the lesion region Ln is present is the block 2A, 1B, 2B,3B, or 2C, the lesion-state analyzing section 36 b determines that thevisibility is high.

When the distance from the center pixel position of the observationimage G1 is outputted as the analysis result, the lesion-state analyzingsection 36 b determines that the visibility is low when the distance islarger than a predetermined distance set in advance. On the other hand,the lesion-state analyzing section 36 b determines that the visibilityis high when the distance is equal to or smaller than the predetermineddistance.

(f) Color and Luminance of Lesion Region Ln

When the color and the luminance of the lesion region Ln outputted fromthe lesion-color/luminance analyzing section 361A6 as the analysisresult are close to a color and luminance of a normal mucous membrane,the lesion-state analyzing section 36 b determines that the visibilityis low. On the other hand, when the color and the luminance of thelesion region Ln are different from the color and the luminance of thenormal mucous membrane, the lesion-state analyzing section 36 bdetermines that the visibility is high. As the color (a color pixelvalue) and the luminance (a luminance value) of the normal mucousmembrane serving as a determination reference, a color and luminanceregistered in advance may be used or values of a normal mucous membraneportion in the observation image G1 in which the lesion region Ln ispresent may be used.

(g) Organ Part where Lesion Region Ln is Located

When the organ part where the lesion region Ln is located outputted fromthe organ-part analyzing section 361A7 as the analysis resultcorresponds to a part with high visibility set in advance, thelesion-state analyzing section 36 b determines that the visibility ishigh. For example, organ parts described below are set as parts withhigh visibility.

When the lesioned part is a large intestine: a descending colon, atransverse colon, and a cecum.

When the lesioned part is a stomach: a cardiac orifice, a stomachfundus, a gastric corpus, a gastric angle, a vestibule, a pyloricregion, a pylorus, and a duodenum.

When the lesioned part is a small intestine: a jejunum and an ileum.

When the lesioned part is an esophagus: a cervical esophagus, a thoracicesophagus, and an abdominal esophagus.

When the organ part where the lesion region Ln is located corresponds toa part with low visibility set in advance, the lesion-state analyzingsection 36 b determines that the visibility is low. For example, organparts described below are set as parts with low visibility.

When the lesioned part is a large intestine: a rectum, a sigmoid colon,a left flexure of colon (a splenic flexure), a right flexure of colon (ahepatic flexure), and an ascending colon.

In other words, when the organ part where the lesion region Ln islocated is a part where lesions frequently occur or a part where anexamination image is hard to see and a lesion tends to be overlooked,the lesion-state analyzing section 36 b determines that the organ partis the part with low visibility.

When only one item is selected as the analysis item of visibility inS11, the lesion-state analyzing section 36 b outputs, as the visibilityof the lesion region Ln, a visibility determination result concerningthe item and ends a series of lesion analysis processing. When two ormore items are selected as the analysis item of visibility in S11, thelesion-state analyzing section 36 b refers to visibility determinationresults of the selected items and determines the visibility of thelesion region Ln.

Examples of a method of determining visibility when a plurality of itemsare selected include a majority method. In other words, when the numberof items determined as having high visibility is larger than the numberof items determined as having low visibility in visibility determinationresults of selected items, the lesion-state analyzing section 36 bdetermines that the visibility of the lesion region Ln is high. On theother hand, when the number of items determined as having highvisibility is equal to or smaller than the number of items determined ashaving low visibility, the lesion-state analyzing section 36 bdetermines that the visibility of the lesion region Ln is low.

Examples of another determination method include a point method. Inother words, according to the visibility, a point is given to each ofthe respective items (a) to (g) explained above. For example, in therespective items, +1 point is given when it is determined that thevisibility is low and −1 point is given when it is determined that thevisibility is high. Note that, concerning items particularlycontributing to the visibility (for example, the item (c) and the item(e)), points weighted compared with the other items may be set to begiven, for example, +3 points are given when it is determined that thevisibility is low and −3 points are given when it is determined that thevisibility is high.

When the visibility determination results of the selected items areconverted into points and a sum of the points is calculated and the sumof the points is larger than a threshold (for example, 1 point) set inadvance, the lesion-state analyzing section 36 b determines that thevisibility of the lesion region Ln is low and outputs a determinationresult. On the other hand, when the sum of the points is equal to orsmaller than the threshold set in advance, the lesion-state analyzingsection 36 b determines that the visibility of the legion region Ln ishigh and outputs a determination result.

The determination result is outputted to the display-extension-timesetting section 36 c. The display-extension-time setting section 36 cselects an appropriate display extension time based on the visibilitydetermination result of the lesion-state analyzing section 36 b. Inother words, the display-extension-time setting section 36 c sets adisplay extension time longer as the visibility is lower. Thedisplay-extension-time setting section 36 c may calculate the displayextension time based on the visibility determination result. Forexample, concerning each of the respective items (a) to (g) explainedabove, an increase or decrease value of display extension time (thenumber of frames) is set in advance according to a detection result. Forexample, concerning the item (d) a size of the lesion region Ln, theincrease or decrease value is set as −2 frames in the case of a largesize, set as ±0 frame in the case of a normal size, and set as +2 framesin the case of a small size. For example, concerning the item (a) adistance between the endoscope 21 and the lesion region Ln, the increaseor decrease value is set as +2 frames when the distance is longer than apredetermined distance range, set as ±0 frame when the distance iswithin a predetermined range, and set as −2 frames when the distance isshorter than the predetermined distance range. In this way, the increaseor decrease values are set for all items. It is also possible tocalculate a sum of increase or decrease values of the display extensiontime (the number of frames) based on the visibility determinationresults of the selected items and set the display extension time.

Note that when the point method is used, the visibility determination inthe respective items (a) to (g) may also be performed by the pointmethod. For example, concerning the item (d), −1 point is given when theposition of the lesion region Ln is present within a range of the centerpixel position of the observation image G1 to a first threshold, 0 pointis given when the position of the lesion region Ln is present within arange of the first threshold to a second threshold, and +1 point isgiven when the position of the lesion region Ln is present within arange of the second threshold to a third threshold (the firstthreshold<the second threshold<the third threshold). A method ofoutputting a given point as a determination result of the visibilityand, in S19, calculating a sum of points outputted as determinationresults of the respective items may be used.

As explained above, according to the embodiment explained above, byadjusting, based on the visibility of the lesion region Ln analyzed bythe lesion-state analyzing section 36 b, the time period (the displayextension time) in which the marker image G2 is continuously displayedafter the detection of the lesion region Ln is interrupted, it ispossible to reduce overlooking of the lesioned part that could occur inthe endoscopic observation and improve the visibility.

Modification of the Second Embodiment

In the second embodiment explained above, the visibility is determinedin a lesion unit. However, in the modification, the visibility isdetermined in an image unit.

FIG. 11 is a flowchart showing an example of a flow of lesion analysisprocessing according to the modification. The visibility analyzingsection 361, in particular, an image-unit-information analyzing section361B in the lesion-state analyzing section 36 b shown in FIG. 9 relatesto the processing shown in FIG. 11 .

The lesion-state analyzing section 36 b analyzes the visibility of thelesion region Ln in the observation image G1. When the visibility isanalyzed in an image unit, a number-of-lesions analyzing section 361B1extracts the number of lesion regions Ln present in the observationimage G1 (S21).

Subsequently, the lesion-state analyzing section 36 b determinesvisibility of the observation image G1 based on the number of lesionregions Ln extracted by the number-of-lesions analyzing section 361B1(S22). In other words, when the number of lesion regions Ln is largerthan a threshold (for example, three) set in advance, the lesion-stateanalyzing section 36 b determines that the visibility is low. On theother hand, when the number of lesion regions Ln is equal to or smallerthan the threshold set in advance, the lesion-state analyzing section 36b determines that the visibility is high.

A determination result is outputted to the display-extension-timesetting section 36 c. The display-extension-time setting section 36 cselects an appropriate display extension time based on the visibilitydetermination result of the lesion-state analyzing section 36 b.

Note that the determination of the visibility may be performed usingboth of the analysis result in an image unit and the analysis result ina lesion unit. In other words, after a series of procedures of S11 toS18 shown in FIG. 10 is performed to analyze the visibility in a lesionunit, a procedure of S21 shown in FIG. 11 is subsequently performed. Thedetermination of the visibility may be comprehensively performed usingan analysis result in a lesion unit and an analysis result in an imageunit acquired by these procedures.

As explained above, according to the modification, it is possible toobtain the same effects as the effects of the respective embodimentsexplained above.

Third Embodiment

In the second embodiment explained above, according to the visibility ofthe lesion region Ln, the time period (the display extension time) inwhich the marker image G2 is continuously displayed after the detectionof the lesion region Ln is interrupted. However, in the presentembodiment, importance of the lesion region Ln is analyzed and a displayextension time is determined based on a result of the analysis.

An image processing apparatus in the present embodiment has the sameconfiguration as the configuration of the image processing apparatus 32in the first embodiment. The same components are denoted by the samereference numerals and signs and explanation of the components isomitted.

FIG. 12 is a flowchart showing an example of a flow of lesion analysisprocessing according to the third embodiment. An importance analyzingsection 362, in particular, a lesion-type analyzing section 362A in thelesion-state analyzing section 36 b shown in FIG. 9 relates to theprocessing shown in FIG. 12 .

The lesion-state analyzing section 36 b analyzes importance of thelesion region Ln. First, the lesion-state analyzing section 36 b selectsan item for which an analysis of importance is performed (S31). Examplesof analysis items of importance include items such as (h) malignancy ofthe lesion region Ln, (i) an organ part where the lesion region Ln islocated, and (j) a color and luminance of the lesion region Ln. Thelesion-state analyzing section 36 b performs an analysis concerningitems selected out of these items. Note that the analysis of importancemay be performed by selecting only one item or may be performed byselecting a plurality of items.

(h) Malignancy of Lesion Region Ln

When this item is selected as the analysis item, processing (S21)explained below is performed. A malignancy analyzing section 362A1 shownin FIG. 9 relates to the processing of S21.

The malignancy analyzing section 362A1 classifies malignancy of thelesion region Ln. The classification of the malignancy is selected by anobservation method. For example, when a narrowband observation isperformed, the malignancy of the lesion region Ln is classified using anexisting malignancy classification such as a NICE (NBI InternationalColorectal Endoscopic) classification or a JNET (The Japan NBI ExpertTeam) classification.

The NICE classification is a simple category classification of threeTypes 1 to 3. Classification is performed from three viewpoints of (1)color tone (color), (2) microvasculature construction (vessels), and (3)surface structure (surface pattern). Type 1 is an indicator ofnon-tumor, Type 2 is an indicator of adenoma to intramucosal carcinoma,and Type 3 is an indicator of SM massively invasive cancer.

In the case of a dye magnifying observation, the malignancy of thelesion region Ln is classified using a PIT pattern classification or thelike.

As explained above, the malignancy analyzing section 362A1 classifiesthe malignancy of the lesion region Ln and outputs the malignancy of thelesion region Ln as an analysis result.

(i) Organ Part where Lesion Region Ln is Located

When this item is selected as the analysis item, processing (S33)explained below is performed. An organ-part analyzing section 362A2shown in FIG. 9 relates to the processing of S33.

The organ-part analyzing section 362A2 performs estimation of anobservation part. For example, when an organ to be observed is a largeintestine, the organ-part analyzing section 362A2 recognizes a rectum, asigmoid colon, a descending colon, a left flexure of colon (a splenicflexure), a transverse colon, a right flexure of colon (a hepaticflexure), an ascending colon, and a caecum. When the organ to beobserved is a stomach, the organ-part analyzing section 362A2 recognizesa cardiac orifice, a stomach fundus, a gastric corpus, a gastric angle,a vestibule, a pyloric region, a pylorus, and a duodenum. In the case ofa small intestine, the organ-part analyzing section 362A2 recognizes ajejunum and an ileum. In the case of an esophagus, the organ-partanalyzing section 362A2 recognizes a cervical esophagus, a thoracicesophagus, and an abdominal esophagus. More specifically, the organ-partanalyzing section 362A2 can perform part (position) estimation using anSVM or the like by collecting image data in which the large intestine,the stomach, the small intestine, and the esophagus are photographed andperforming machine learning using the image data.

As explained above, the organ-part analyzing section 362A2 estimates anobservation part and outputs the observation part as an analysis result.Note that processing content of the organ-part analyzing section 362A2is the same as the processing content of the organ-part analyzingsection 361A7. Therefore, the result of the organ-part analyzing section361A7 may be used.

(j) Color and Luminance of Lesion Region Ln

When this item is selected as the analysis item, processing (S34)explained below is performed. A lesion-color/luminance analyzing section362A3 shown in FIG. 9 relates to the processing of S34.

When the observation image G1 is an image formed by three components ofred (R), green (G), and blue (B), the lesion-color/luminance analyzingsection 362A3 extracts pixel values (R pixel values, G pixel values, andB pixel values) of respective pixels included in the lesion region Ln.The lesion-color/luminance analyzing section 362A3 calculates an averageof each of the R pixel values, the G pixel values, and the B pixelvalues and sets the average as a color pixel value of the lesion regionLn. Note that other statistical values such as a mode may be used forcalculation of a pixel value of the lesion region Ln rather than theaverage.

The lesion-color/luminance analyzing section 362A3 extracts luminancevalues of the respective pixels included in the lesion region Ln,calculates an average value of the luminance values, and sets theaverage value as a luminance value of the lesion region Ln. Note thatother statistical values such as a mode may be used for calculation of aluminance value of the lesion region Ln rather than the average.

As explained above, the lesion-color/luminance analyzing section 362A3calculates the color pixel value and the luminance value of the lesionregion Ln in the observation image G1 and outputs the color pixel valueand the luminance value as an analysis result. Note that processingcontent of the lesion-color/luminance analyzing section 362A3 is thesame as the processing content of the lesion-color/luminance analyzingsection 361A6. Therefore, the processing result of thelesion-color/luminance analyzing section 361A6 may be used.

When processing of selected one or more items among the processing ofS32 to the processing of S34 ends, the lesion-state analyzing section 36b determines importance based on analysis results of these items (S35).First, the lesion-state analyzing section 36 b determines importance foreach of the analysis items.

(h) Malignancy of Lesion Region Ln

When the malignancy of the lesion region Ln outputted from themalignancy analyzing section 362A1 as the analysis result corresponds toa category set in advance, the lesion-state analyzing section 36 bdetermines that the importance is high.

(i) Organ Part where Lesion Region Ln is Located

When the organ part where the lesion region Ln is located outputted fromthe organ-part analyzing section 362A2 as the analysis resultcorresponds to a part with high importance set in advance, thelesion-state analyzing section 36 b determines that the importance ishigh. For example, organ parts described below are set as parts withhigh importance.

When the lesioned part is a large intestine: a sigmoid colon.

When the organ part where the lesion region Ln is located is other thanthe part described above, the lesion-state analyzing section 36 bdetermines that the importance is low. In other words, in the case of apart where a risk of worsening of a symptom is high if unattended, thelesion-state analyzing section 36 b determines the part as a part withhigh importance. Note that the organ part where the lesion region Ln islocated is also described as the determination item of the visibility.However, a level of the visibility and a level of the importance arerespectively determined by independent evaluation indicators.

(j) Color and Luminance of Lesion Region Ln

When the color and luminance of the lesion region Ln outputted from thelesion-color/luminance analyzing section 362A3 as the analysis resultare close to a color and luminance of a lesioned part with highimportance registered in advance, the lesion-state analyzing section 36b determines that the importance is high. Note that the color andluminance of the lesion region Ln are also described as thedetermination item of the visibility. However, a level of the visibilityand a level of the importance are respectively determined by independentevaluation indicators.

When only one item is selected as the analysis item of importance inS31, the lesion-state analyzing section 36 b outputs an importancedetermination result concerning the item as the importance of the lesionregion Ln and ends a series of lesion analysis processing. When two ormore items are selected as the analysis item of importance in S31, thelesion-state analyzing section 36 b refers to importance determinationresults of the selected items and determines the importance of thelesion region Ln.

As a method of determining importance when a plurality of items areselected, the majority method and the point method can be used as in thecase of the determination method for visibility.

The determination result is outputted to the display-extension-timesetting section 36 c. The display-extension-time setting section 36 cselects an appropriate display extension time based on the importancedetermination result of the lesion-state analyzing section 36 b. Inother words, the display-extension-time setting section 36 c sets thedisplay extension time longer as the importance is higher.

As explained above, according to the embodiment explained above, byadjusting, based on the importance of the lesion region Ln analyzed bythe lesion-state analyzing section 36 b, the time period (the displayextension time) in which the marker image G2 is continuously displayedafter the detection of the lesion region Ln is interrupted, it ispossible to reduce overlooking of the lesioned part that could occur inthe endoscopic observation and improve the visibility.

Modification of the Third Embodiment

In the third embodiment explained above, the importance is determinedbased on the type of the lesion. However, in the modification, theimportance is determined according to a shape and a size of the lesion.

FIG. 13 is a flowchart showing an example of a flow of lesion analysisprocessing according to the modification. The importance analyzingsection 362, in particular, a lesion-shape/size analyzing section 362Bin the lesion-state analyzing section 36 b shown in FIG. 9 relates tothe processing shown in FIG. 13 .

The lesion-state analyzing section 36 b analyzes the importance of thelesion region Ln. First, the lesion-state analyzing section 36 b selectsan item for which an analysis of importance is performed (S41). Examplesof analysis items of importance concerning a shape and a size of alesion include items such as (k) a shape of the lesion region Ln and (l)a size of the lesion region Ln. The lesion-state analyzing section 36 bperforms an analysis concerning items selected out of these items. Notethat the analysis of the importance may be performed by selecting onlyone item or may be performed by selecting a plurality of items.

(k) Shape of Lesion Region Ln

When this item is selected as the analysis item, processing (S42)explained below is performed. A lesion-shape analyzing section 362B1shown in FIG. 9 relates to the processing of S42.

The lesion-shape analyzing section 362B1 performs identificationclassification based on a shape of a lesioned part. More specifically,the lesion-shape analyzing section 362B1 creates a mask image indicatinga lesion region and calculates a shape feature value based on the image.The shape feature value is classified into, using a classifier such asan SVM, one of a plurality of classes generated by machine learning. Asthe shape feature value, a publicly-known parameter such as circularity,moment, or fractal dimension is used.

For example, in the case of a large intestine polyp, there are anelevated type (I type) and a superficial type (II type). As the elevatedtype, there are sessile (Is) without a constriction in a rising part,sub-sessile (Isp) with a constriction in a rising part, and pedunculate(Ip) with a peduncle. In the superficial type, the large intestine polypis classified into an elevated type (IIa), a flat type (IIb), and adepressed type (IIc).

For example, in the case of a stomach polyp, there are a submucosaltumor type (an elevated I type), a sessile type (an elevated II type), asub-sessile type (an elevated III type), and a pedunculate type (anelevated IV type). For example, in the case of a stomach cancer, thestomach cancer is classified into a superficial type (a 0 type), atumorous type (a 1 type), an ulcerative and localized type (a 2 type),an infiltrative ulcerative type (a 3 type), and a diffuse infiltrativetype (a 4 type), and the like.

As explained above, the lesion-shape analyzing section 362B1 identifiesa shape of the lesioned part and outputs the shape as an analysisresult. Note that processing content of the lesion-shape analyzingsection 362B1 is the same as the processing content of the lesion-shapeanalyzing section 361A3. Therefore, the result of the lesion-shapeanalyzing section 361A3 may be used.

(i) Size of Lesion Region Ln

When this item is selected as the analysis item, processing (S43)explained below is performed. A lesion-size estimating section 362B2shown in FIG. 9 relates to the processing of S43.

First, the lesion-size estimating section 362B2 estimates image pickupdistances to respective pixels in an image. The lesion-size estimatingsection 362B2 may perform the estimation of the image pickup distancesusing the method explained above or the like. The lesion-distanceestimating section 361A1 may perform the processing and acquire theresult.

Subsequently, the lesion-size estimating section 362B2 provides athreshold smaller than an image pickup distance of a pixel near a lesionand a threshold larger than the image pickup distance and extracts,through processing by using the thresholds, a region of an image pickupdistance zone where the lesion is present. The lesion-size estimatingsection 362B2 calculates circularity of the region and, when thecircularity is larger than a predetermined value, detects the region asa lumen.

Finally, the lesion-size estimating section 362B2 compares the lumen andthe lesioned part and estimates a size of the lesioned part.

More specifically, the lesion-size estimating section 362B2 estimates anactual size of the lesion by calculating a ratio occupied by length ofthe lesion with respect to a circumferential length of the detectedlumen. Note that it is also possible to set circumferential lengths oflumens in respective organ parts (positions) beforehand based on anatomyand improve accuracy of size estimation. For example, in the case of alarge intestine examination, estimation of a part (position) of alesioned part of a large intestine may be performed based on aninsertion amount of the insertion section to improve the accuracy of thesize estimation of the actual size of the lesion based on the ratiooccupied by the length of the lesion with respect to the circumferentiallength of the lumen set beforehand in the part (position) of theestimated lesioned part.

As explained above, the lesion-size estimating section 362B2 estimatesthe size of the lesioned part in comparison with the circular size ofthe lumen photographed in the endoscopic image and outputs the size ofthe lesioned part as an analysis result. Note that processing content ofthe lesion-size estimating section 362B2 is the same as the processingcontent of the lesion-size estimating section 361A4. Therefore, theresult of the lesion-size estimating section 361A4 may be used.

When processing of selected one or more items of the processing of S42and the processing of S43 ends, the lesion-state analyzing section 36 bdetermines importance based on analysis results of the items (S44).First, the lesion-state analyzing section 36 b determines importance foreach of analysis items.

(k) Shape of Lesion Region Ln

When the shape of the lesion region Ln outputted from the lesion-shapeanalyzing section 362B1 as the analysis result corresponds to a shapewith high importance set in advance, the lesion-state analyzing section36 b determines that the importance is high. For example, shapesdescribed below are set as the shape with high importance.

When the lesioned part is a large intestine: a superficial flat type(IIb) and a superficial depressed type (IIc).

When the lesioned part is a stomach: a tumorous type (a 1 type), anulcerative and localized type (a 2 type), an infiltrative ulcerativetype (a 3 type), and a diffuse infiltrative type (a 4 type).

When the shape of the lesion region Ln is other than the shapesdescribed above, the lesion-state analyzing section 36 b determines thatthe importance is low. In other words, in the case of the shape of thelesion region Ln with a high risk of worsening of a symptom ifunattended, the lesion-state analyzing section 36 b determines thelesion region Ln as a part with high importance. Note that the shape ofthe lesion region Ln is also described as the determination item of thevisibility. However, a level of the visibility and a level of theimportance are respectively determined by independent evaluationindicators.

(l) Size of Lesion Region Ln

When the size of the lesion region Ln outputted from the lesion-sizeestimating section 362B2 as the analysis result is equal to or smallerthan a predetermined size (for example, 5 mm) set in advance, thelesion-state analyzing section 36 b determines that the importance islow. On the other hand, when the size of the lesion region Ln is largerthan the predetermined size, the lesion-state analyzing section 36 bdetermines that the importance is high. Note that the size of the lesionregion Ln is also described as the determination item of the visibility.However, a level of the visibility and a level of the importance arerespectively determined by independent evaluation indicators.

As a method of determining importance when a plurality of items areselected, the majority method and the point method can be used as in thecase of the determination method for visibility.

The determination result is outputted to the display-extension-timesetting section 36 c. The display-extension-time setting section 36 cselects an appropriate display extension time based on the importancedetermination result of the lesion-state analyzing section 36 b. Inother words, the display-extension-time setting section 36 c sets thedisplay extension time longer as the importance is higher.

Note that the determination of the importance may be performed usingboth of the analysis result based on the lesion type and the analysisresult based on the lesion shape and size. In other words, after aseries of procedures of S31 to S34 shown in FIG. 12 is performed toperform the analysis of the importance based on the lesion type, theprocedures of S41 to S43 shown in FIG. 13 are subsequently performed.The determination of the importance may be comprehensively performedusing the analysis result based on the lesion type and the analysisresult based on the lesion shape and size acquired by these procedures.

A state of a lesion may be determined using both of the analysis resultof the visibility and the analysis result of the importance.

As explained above, according to the embodiments and the modificationsexplained above, it is possible to provide an image processing apparatusfor endoscope that can reduce overlooking of a lesioned part that couldoccur in an endoscopic observation and improve visibility by adjusting,according to a state of the lesion region Ln, a time period (a displayextension time) in which the marker image G2 is continuously displayedafter the detection of the lesion region Ln is interrupted.

The present invention is not limited to the embodiments explained above.It goes without saying that various changes and applications arepossible within a range not departing from the gist of the invention.

What is claimed is:
 1. An image processing apparatus comprising: aprocessor configured to: receive an observation image obtained bypicking up an image of an object with an endoscope; detect one or morelesioned parts, which each is an observation target of the endoscope,from the observation image; perform highlighting processing on each ofthe one or more lesioned parts; analyze visibility of each of the one ormore lesioned parts; analyze an importance of the each of the one ormore lesioned parts; set a display extension time of the highlightingprocessing of each of the one or more lesioned parts according to theanalyzed visibility and the analyzed importance; and output theobservation image to which each of the one or more lesioned parts onwhich the highlighting processing has been performed is added.
 2. Theimage processing apparatus according to claim 1, wherein the processoris configured to: estimate a distance of the each of the one or morelesioned parts from the endoscope; and analyze the visibility of theeach of the one or more lesioned parts based on the distance of the eachof the one or more lesioned parts from the endo scope estimated.
 3. Theimage processing apparatus according to claim 1, wherein the processoris configured to: calculate an occupied area of the each of the one ormore lesioned parts in the observation image; and analyze the visibilityof the each of the one or more lesioned parts based on the occupied areaof the each of the one or more lesioned parts in the observation imagecalculated.
 4. The image processing apparatus according to claim 1,wherein the processor is configured to: analyze a shape of the each ofthe one or more lesioned parts; and analyze the visibility of the eachof the one or more lesioned parts based on the shape of the each of theone or more lesioned parts analyzed.
 5. The image processing apparatusaccording to claim 4, wherein, when an observation target organ part ofthe endoscope is a large intestine, the processor is configured to:analyze which shape of sessile, sub-sessile, pedunculate, a superficialelevated type, a superficial flat type, and a superficial depressed typethe shape of the each of the one or more lesioned parts is; and analyzethe visibility of the each of the one or more lesioned parts based onthe shape of sessile, sub-sessile, pedunculate, the superficial elevatedtype, the superficial flat type, and the superficial depressed type theshape of the each of the one or more lesioned parts is analyzed to be,and wherein, when the observation target organ part is a stomach, theprocessor is configured to: analyze which shape of a submucosal tumortype, a sessile type, a sub-sessile type, a pedunculate type, asuperficial type, a tumorous type, an ulcerative and localized type, aninfiltrative ulcerative type, and a diffuse infiltrative type the shapeof the each of the one or more lesioned parts is; and analyze thevisibility of the each of the one or more lesioned parts based on theshape of the submucosal tumor type, the sessile type, the sub-sessiletype, the pedunculate type, the superficial type, the tumorous type, theulcerative and localized type, the infiltrative ulcerative type, and thediffuse infiltrative type the shape of the each of the one or morelesioned parts is analyzed to be.
 6. The image processing apparatusaccording to claim 1, wherein the processor is configured to: estimate asize of the each of the one or more lesioned parts; and analyze thevisibility of the each of the one or more lesioned parts based on thesize of the each of the one or more lesioned parts estimated.
 7. Theimage processing apparatus according to claim 1, wherein the processoris configured to: analyze a position of the each of the one or morelesioned parts; and analyze the visibility of the each of the one ormore lesioned parts based on the position of the each of the one or morelesioned parts analyzed.
 8. The image processing apparatus according toclaim 1, wherein the processor is configured to: analyze at least one ofa color or luminance of the each of the one or more lesioned parts; andanalyze the visibility of the each of the one or more lesioned partsbased on the at least one or of the color or luminance of the each ofthe one or more lesioned parts analyzed.
 9. The image processingapparatus according to claim 1, wherein the processor is configured to:analyze an organ part where the each of the one or more lesioned partsis located; and analyze the visibility of the each of the detected oneor more lesioned parts based on the organ part analyzed.
 10. The imageprocessing apparatus according to claim 9, wherein, when the organ partis a large intestine, the processor is configured to: estimate which ofa rectum, a sigmoid colon, a descending colon, a left flexure of colon,a transverse colon, a right flexure of colon, an ascending colon, and acaecum the each of the one or more lesioned parts is located; andanalyze the visibility of the each of the one or more lesioned partsbased on the rectum, the sigmoid colon, the descending colon, the leftflexure of colon, the transverse colon, the right flexure of colon, theascending colon and the caecum that the each of the one or more lesionedparts is estimated to be located, wherein, when the organ part is astomach, the processor is configured to: estimate which of a cardiacorifice, a stomach fundus, a gastric corpus, a gastric angle, avestibule, a pyloric region, a pylorus, and a duodenum the each of theone or more lesioned parts is located; and analyze the visibility of theeach of the one or more lesioned parts based on the cardiac orifice, thestomach fundus, the gastric corpus, the gastric angle, the vestibule,the pyloric region, the pylorus and the duodenum that the each of theone or more lesioned parts is estimated to be located, wherein, when theorgan part is a small intestine, the processor is configured to:estimate which of a jejunum and an ileum the each of the one or morelesioned parts is located; and analyze the visibility of the each of theone or more lesioned parts based on the jejunum and the ileum that theeach of the one or more lesioned parts is estimated to be located, andwherein, when the organ part is an esophagus, the processor isconfigured to: estimate which of a cervical esophagus, a thoracicesophagus, and an abdominal esophagus the each of the one or morelesioned parts is located; and analyze the visibility of the each of theone or more lesioned parts based on the cervical esophagus, the thoracicesophagus, and the abdominal esophagus that the each of the one or morelesioned parts is estimated to be located.
 11. The image processingapparatus according to claim 1, wherein the processor is configured to:analyze a number of the one or more lesioned parts in the observationimage; and analyze the visibility of the each of the one or morelesioned parts based on the number of the one or more lesioned partsanalyzed.
 12. The image processing apparatus according to claim 1,wherein the processor is configured to: set the display extension timeto be a first length when it is determined that the visibility is higherthan a predetermined threshold; and set the display extension time to bea second length longer than the first length when it is determined thatthe visibility is lower than the predetermined threshold.
 13. The imageprocessing apparatus according to claim 1, wherein the processor isconfigured to: analyze a type of the each of the one or more lesionedparts; and analyze the importance of the each of the one or morelesioned parts based on the type of the each of the one or more lesionedparts analyzed.
 14. The image processing apparatus according to claim 1,wherein the processor is configured to: analyze malignancy of the eachof the one or more lesioned parts; and analyze the importance of theeach of the one or more lesioned parts based on the malignancy of theeach of the one or more lesioned parts analyzed.
 15. The imageprocessing apparatus according to claim 1, wherein the processor isconfigured to: analyze an organ part, where the each of the one or morelesioned parts is located; and analyze the importance of the each of theone or more lesioned parts based on the organ part analyzed.
 16. Theimage processing apparatus according to claim 15, wherein, when theorgan part is a large intestine, the processor is configured to:estimate which of a rectum, a sigmoid colon, a descending colon, a leftflexure of colon, a transverse colon, a right flexure of colon, anascending colon, and a caecum the each of the one or more lesioned partsis located; and analyze the importance of the each of the one or morelesioned parts based on the rectum, the sigmoid colon, the descendingcolon, the left flexure of colon, the transverse colon, the rightflexure of colon, the ascending colon and the caecum that the each ofthe one or more lesioned parts is estimated to be located, wherein, whenthe organ part is a stomach, the processor is configured to: estimatewhich of a cardiac orifice, a stomach fundus, a gastric corpus, agastric angle, a vestibule, a pyloric region, a pylorus, and a duodenumthe each of the one or more lesioned parts is located; and analyze theimportance of the each of the one or more lesioned parts based on thecardiac orifice, the stomach fundus, the gastric corpus, the gastricangle, the vestibule, the pyloric region, the pylorus and the duodenumthat the each of the one or more lesioned parts is estimated to belocated, wherein, when the organ part is a small intestine, theprocessor is configured to: estimate which of whether the organ part isa jejunum and an ileum the each of the one or more lesioned parts islocated; and analyze the importance of the each of the one or morelesioned parts based on the jejunum and the ileum that the each of theone or more lesioned parts is estimated to be located, and wherein, whenthe organ part is an esophagus, the processor is configured to: estimatewhich of a cervical esophagus, a thoracic esophagus, and an abdominalesophagus the each of the one or more lesioned parts is located; andanalyze the importance of the each of the one or more lesioned partsbased on the cervical esophagus, the thoracic esophagus, and theabdominal esophagus that the each of the one or more lesioned parts isestimated to be located.
 17. The image processing apparatus according toclaim 1, wherein the processor is configured to: analyzes a color andluminance of the each of the one or more lesioned parts; and analyze theimportance of the each of the one or more lesioned parts based on thecolor and luminance of the each of the one or more lesioned partsanalyzed.
 18. The image processing apparatus according to claim 1,wherein the processor is configured to analyze the importance of theeach of the one or more lesioned parts based on a shape and a size ofthe each of the one or more lesioned parts.
 19. The image processingapparatus according to claim 18, wherein the processor is configured toanalyze the importance based on the shape of the each of the each of theone or more lesioned parts.
 20. The image processing apparatus accordingto claim 19, wherein, when an observation target organ part of theendoscope is a large intestine, the processor is configured to: analyzewhich shape of sessile, sub-sessile, pedunculate, a superficial elevatedtype, a superficial flat type, and a superficial depressed type theshape of the each of the detected one or more lesioned parts is; andanalyze the importance of the each of the one or more lesioned partsbased on the shape of sessile, sub-sessile, pedunculate, the superficialelevated type, the superficial flat type, and the superficial depressedtype the shape of the each of the one or more lesioned parts is analyzedto be, and wherein, when the observation target organ part is a stomach,the processor is configured to: analyze which shape of a submucosaltumor type, a sessile type, a sub-sessile type, a pedunculate type, asuperficial type, a tumorous type, an ulcerative and localized type, aninfiltrative ulcerative type, and a diffuse infiltrative type the shapeof the each of the one or more lesioned parts is; and analyze theimportance of the each of the one or more lesioned parts based on theshape of the submucosal tumor type, the sessile type, the sub-sessiletype, the pedunculate type, the superficial type, the tumorous type, theulcerative and localized type, the infiltrative ulcerative type, and thediffuse infiltrative type the shape of the each of the one or morelesioned parts is analyzed to be.
 21. The image processing apparatusaccording to claim 18, wherein the processor is configured to analyzethe importance based on the size of the each of the one or more lesionedparts.
 22. The image processing apparatus according to claim 1, whereinthe processor is configured to: set the display extension time to be afirst length when it is determined that the importance is higher than apredetermined threshold; and set the display extension time to be asecond length shorter than the first length when it is determined thatthe importance is lower than the predetermined threshold.
 23. The imageprocessing apparatus according to claim 1, wherein the processor isconfigured to analyze, as analysis of the visibility, at least one of adistance from the endoscope, an occupied area, a shape, a size, aposition in the observation image, a color, luminance or an organ part.24. The image processing apparatus according to claim 1, wherein theprocessor is configured to sequentially receive a plurality of theobservation image.
 25. The image processing apparatus according to claim1, wherein the processor is configured to perform extension of thehighlighting processing for the display extension time after detectionof the one or more lesioned parts became impossible.
 26. An imageprocessing method comprising: receiving an observation image obtained bypicking up an image of an object with an endoscope; detecting one ormore lesioned parts, which each is an observation target of theendoscope, from the observation image; performing highlightingprocessing on each of the one or more lesioned parts; analyzingvisibility of each of the one or more lesioned parts; analyzing animportance of the each of the one or more lesioned parts; setting adisplay extension time of the highlighting processing of each of the oneor more lesioned parts according to the analyzed visibility and theanalyzed importance; and outputting the observation image to which eachof the one or more lesioned parts on which the highlighting processinghas been performed is added.
 27. A non-transitory computer-readablerecording medium that stores a computer program, the computer programcausing a computer to: receive an observation image obtained by pickingup an image of an object with an endoscope; detect one or more lesionedparts, which each is an observation target of the endoscope, from theobservation image; perform highlighting processing on each of the one ormore lesioned parts; analyze visibility of each of the one or morelesioned parts; analyze an importance of the each of the one or morelesioned parts; set a display extension time of the highlightingprocessing of each of the one or more lesioned parts according to theanalyzed visibility and the analyzed importance; and output theobservation image to which each of the one or more lesioned parts onwhich the highlighting processing has been performed is added.